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37 CFR § 1.91 PETITION 
TO MAKE EXHIBITS ACCOMPANYING 37 CFR §§ 1.131 and 1.132 AFFIDAVIT A 

PART OF THE OFFICIAL RECORD 



37 CFR § 1.91(a) provides that "[a] model or exhibit will not be admitted as part of the 
record of an application unless it: (1) substantially conforms to the requirements of § 1.52 or § 
1.84; (2) is specifically required by the Office; or (3) is filed with a petition under this section 
including: (i) the fee set forth in § 1.17(h); and (ii) an explanation of why entry of the model or 
exhibit in the file record is necessary to demonstrate patentability." 

Applicants have submitted an Affidavit of David M. Tumey under 37 CFR §§ 1.131 and 
1.132 in an effort to swear behind a cited reference or alternatively to establish evidence of non- 
obviousness. Patent Office practice under 37 CFR § 1.131 requires Applicants to attach 
documentary evidence of their diligent efforts to reduce their invention to practice. Because 
much of the Applicants' evidence consists of executable software modules that have evidence of 
their "creation" dates, Applicants have submitted a CD containing those modules as part of 
Exhibit 8 to their Affidavit. Applicants seek to have this evidence entered into the record to 
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establish their diligent efforts to reduce their invention to practice. Furthermore, Applicants have 
submitted a CD as Exhibit 14 which demonstrates actual reduction to practice of the claimed 
invention. This second CD also establishes evidence of non-obviousness and the patentable 
worthiness of the invention. Applicants also wish to have these items entered as part of the 
permanent record for purposes of relying on them, if necessary, in an appeal. 

Accordingly, Applicants respectfully petition the Commissioner to waive the 
requirements of 37 CFR § 1.91 as it pertains to the exhibits to Applicants' Affidavit. 

As set forth in the accompanying fee transmittal form, the Commissioner, is authorized to 
deduct any fees that may be required from Gunn & Lee's deposit account no. 500808. 

Respectfully submitted, 




Eric W. Cernyar 
Reg. No. 45,919 
Gunn & Lee, P.C. 
700 N. St. Mary's St. Ste 1500 
San Antonio, TX 78205 
(210) 886-9500 (phone) 
(210) 886-9883 (fax) 
(210) 863-0063 (cell phone) 
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REMARKS: 

Status of claims: 

Claims 1-16 stand rejected as obvious over U.S. Patent No. 6,175,772 to Kamiya 
et al., for the same reasons set forth in the Examiner's January 30, 2004, and July 6, 
2004, Office Actions. Applicants again respectfully traverse these rejections with both 
argument and affidavit evidence . 

Applicants would note that they have twice amended the claims to make a 
distinction between mere "facial expression" recognition and "biometric" identification. 
In response to the January 30, 2004, Office Action, Applicants amended claims 1,11, and 
12 to add the following limitations: 

Claim 1 : "of a particular one of said animate or inanimate objects." 

Claim 11: "of a particular person" 

Claim 12: "indicative of a particular person." 

In response to the July 6, 2004, Office Action, Applicants amended claims 1 and 12 even 
further to add the following limitations: 

Claim 1: "operable to biometrically identify an imaged one of a plurality of 
animate or inanimate objects having facial or face-like characteristics by 
measuring the facial or face-like characteristics of the imaged object" and 
"biometric" 

Claim 12: "biometrically" 

U.S. Patent No. 6,175,772 to Kamiya et al does not disclose or teach biometric 
facial recognition. Nevertheless, the Examiner argues that "the claim limitations are 
given their broadest reasonable interpretations" 1 and maintains that Kamiya teaches 
"facial expression 'facial recognition'" as claimed. 

Applicants respectfully urge that in light of the repeated amendments and 
arguments that Applicants have made regarding the claims, the claims cannot reasonably 
be given an interpretation that merely covers "facial expression" recognition without any 
capability of biometric identification. 

Furthermore, the attached 37 §§ 1.131/1.132 Affidavit of David M. Tumey (with 
sworn verification by co-inventor Dr. Tony Xu) shows that biometric identification is not 
a trivial or obvious extension of mere face-expression recognition. Paragraph 24 notes 
that the ability to ascertain emotive facial expressions takes less processing than the 
ability to uniquely identify a person. In paragraph 25, Tumey and Dr. Xu attested that 

1 Applicants take this to mean that the Examiner construes claims 1-16 as reading on a 
toy that is capable only of recognizing "facial expressions." 
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"unique facial image recognition is significantly more complex than detecting a not- 
necessarily-unique facial expression." 2 Even the Kamiya reference makes this apparent - 
Kamiya's system only recognized seven basic emotional models - neutral, disgusted, 
happy, sad, surprised, angry, and fearful Col. 6, lines 32-35; col. 7, lines 37-40. 

One of the most significant challenges Tumey and Dr. Xu faced in reducing their 
invention to practice was processing delays. In paragraph 18 of their Affidavit, Tumey 
and Dr. Xu state that "[f]ace recognition algorithms that we had developed earlier for 
security applications were not fast enough to be used in a toy. They typically had about a 
3 -second delay between the image capture and the facial recognition - this delay was too 
long for a toy. Our goal was to make a toy that would recognize and respond to human 
users in real time." Also, Tumey and Dr. Xu's patent specification notes, on page 2, lines 

8- 9, that "identification processing delays can be excessive and unacceptable for many 
applications." And Tumey and Dr. Xu's survey of the prior art revealed that prior 
"approaches to providing an encoded facial image that could be stored, retrieved and 
compared, automatically or manually, at some later time for recognizing said human 
user" were not "viable for use in an articulated and animated toy or video game." Page 4, 
lines 20-24. Tumey and Dr. Xu's affidavit, attached, shows that they spent a great deal of 
time trying to develop efficient yet effective facial-image-recognition algorithms suitable 
for use with an interactive entertainment device. 

Given these challenges and the limitations of the prior art, it cannot be maintained 
that the prior art provided any teaching, suggestion, or motivation (together with a 
reasonable expectation of success) to modify Kamiya, or combine it with other 
references, in order to read on the claims. 

In any event, Tumey and Dr. Xu conceived of the claimed invention before the 
applicable section 102(e) date (April 13, 1998) of the Kamiya reference. See Affidavit fflf 

9- 15. They also diligently reduced their invention to practice. See Affidavit 16-48. 
Applicants offer the attached 37 §§ 1.131/1.132 Affidavit of David M. Tumey (with 
sworn verification by co-inventor Dr. Tony Xu) in order to swear behind the Kamiya 
reference . 

The undersigned is acutely aware of the Patent and Trademark Office's reluctance 
to allow patents in the business method/computer software/biometric 
identification/encryption field. Indeed, the PTO's own statistics show that allowances of 
patents in these fields have slowed down to a crawl. The undersigned presumes that 
much of that hostility derives not only from a spate of negative publicity, but also from 
the plethora of "concept patent applications" that have been filed in these technology 

2 See also Affidavit, at paragraph 28 ("Biometric identification, which again is 
completely different from feature/expression recognition, involves treating the entire face 
as a collection of unique features that are specific to each given individual, i.e., the 
distance between the eyes, nose and mouth coupled with the morphology of each feature 

....")• 
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areas. Many companies have filed applications with little more than the "germ" of an 
idea about a future technology without ever making any attempt to reduce their claimed 
invention to actual practice. 

Tumey's and Dr. Xu's invention is not a mere "concept." With years of effort, 
they reduced their invention to practice . Attached as Exhibit 14 to their Affidavit is a 
copy of a CD with a video demonstration of a working prototype of a toy bear with facial 
image recognition abilities. This video is also available at 

http://www.cernvar.com/toy.mpg . Tumey's and Dr. Xu's efforts, and their willingness to 
share their inventive concepts with the world (through their application) are the kind of 
activities Congress intended the patent system to reward. 

Applicants respectfully ask that the Examiner consider the attached Affidavit and 
Exhibits and withdraw the rejections . A petition under § 1.91 to make the exhibits a part 
of the record accompanies this response. 



Believing that all things raised in the Examiner's December 8, 2004, Office 
Action have been addressed, the undersigned respectfully requests that the application be 
allowed and passed to issue. 

As set forth in the accompanying petition for extension of time, the 
Commissioner, is authorized to deduct any additional fees (beyond the submitted check) 
that may be required from Gunn & Lee's deposit account no. 500808. 



Conclusion 



Respectfully submitted, 



Eric W. Cernyar 
Reg. No. 45,919 
Gunn & Lee, P.C. 
700 N. St. Mary's St. Ste 1500 
San Antonio, TX 78205 
(210) 886-9500 (phone) 
(210) 886-9883 (fax) 
(210) 863-0063 (cell phone) 
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AFFIDAVIT OF DAVID M. TUMEY 
UNDER 37 CFR SS L131 and 1,132 

I, David Malcolm Tumey, being duly sworn, state as follows: 

1 . I am over 2 1 years of age and am competent to make this affidavit. 

2. I graduated with a Bachelor of Science degree in Electrical Engineering 
from the University of Massachusetts in 1985. 

3. I have close to twenty years experience working in the field of electrical 
engineering and electro-mechanical research and design, including extensive experience 
in the design and development of both hard-wired circuits and microcontrollers. 

4. Between 1988 and 1992, 1 worked for the U.S. Air Force as an electrical 
engineer researching advanced cockpit design and aircraft avionic systems. I contributed 
to the development of neural-network based control systems for flight simulators that, by 
detecting a pilot's brain waves and learning associations between the brain waves and 
movements, enabled pilots to fly the simulator with their thoughts. 

5. Since 1992, I have worked as an employee or consultant for various 
medical device firms helping them develop hardware and software control systems for 
sophisticated electromechanical medical devices. 
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6. I am listed as the sole or joint inventor of at least 32 issued U.S. patents 
and dozens of other foreign patents and foreign and domestic patent applications. 

7. I and Dr. Tianning (Tony) Xu are the inventors of the subject matter set 
forth in U.S. Patent Application No. 09/488,390 entitled "Animated Toy Utilizing 
Artificial Intelligence and Facial Image Recognition.' 5 

8. The invention disclosed in 09/488,390 was a difficult and complex 
invention. We expended many years of effort developing the algorithms, methods, 
electrical circuits, and mechanical systems necessary to reduce the invention to actual 
practice. 

Conception of the Invention 

9. In 1989, I began sketching out the structure of an algorithm for 
recognizing facial images. See Exhibit "1." Early on, I realized that some of the biggest 
challenges I needed to overcome involved locating a human or human-like face in an 
image and scaling, rotating, and centering the face in an image window. Id 

10. Over the next several years, Dr. Xu and I began developing algorithms to 
preprocess the images. I developed a crude software program using WATCOM C++ to 
implement some of the algorithms disclosed in my Jan. 5, 1990 notebook entry. See 
Exhibits." 

11. In a 1991 notebook entry I record techniques that we developed for 
finding a face in an image. See Exhibit "4." The algorithms involved complex matrix 
manipulations. Unfortunately, existing computer hardware technology at the time was 
simply inadequate to perform these manipulations in a time-, energy-, and space-efficient 
manner. We also began developing a library of functions to use for future prototypes. 
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12. As development continued, we identified other challenges to successfully 
reducing the face recognition concept to practice. See Exhibit "3." Aside from locating a 
face in an image, the system needed to be capable of determining whether a face even 
existed in the image. Id Furthermore, the system needed to be able to filter the image to 
remove background information. Id 

13. Two notebook entries, one of which is dated Jan. 12, 1992, detail a 
method we developed for enrolling and creating templates for use in later recognition 
efforts. See Exhibits "5" & "6." 

14. A Nov. 12, 1996 notebook entry records my conception of using the face 
recognition methods we previously developed to control the animation of a toy or video 
game or even to direct internet browsing. See Exhibit "7." In that notebook entry, I 
suggest using those methods not only for biometric facial identification, but also for 
facial expression recognition. I also noted that the combined biometric facial recognition 
and facial expression recognition could be accomplished by storing templates 
("eigenfaces") of the same individual with different facial expressions. 

15. These exhibits establish that we had possession of the whole invention 
claimed or something falling within the claim. Exhibit "7" shows entertainment devices, 
including toys, positionable in proximity to animate or inanimate objects. Exhibits "1" 
through "7" show possession of the image capturing and recognition elements (including 
a processor) of claims 1,11, and 12. Exhibit "7" also shows possession of the concept of 
having the device provide entertaining interaction in response to recognizing a human 
face, and in particular, in response to recognizing a human face with a particular 
emotional expression. 
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Diligent Reduction to Practice 

16. From September 1996 until the date we filed our provisional application, 
Dr. Xu and I worked diligently to reduce the invention to practice. Most of this time was 
spent on developing software algorithms to improve the performance of the face 
recognition technology. Because the effort was focused mainly on software, we have 
only a few engineering notebook entries to document the progress. We do, however, 
have several software programs, files, and data to corroborate the statements made in the 
following paragraphs. Attached as Exhibit "8" to this declaration is a four-page table 
listing the functions of several different versions of software programs and executables, 
along with the dates the files were last modified. The actual software files themselves are 
also included in a CD that accompanies Exhibit "8." 

17. In addition, during this timeframe we simultaneously experimented with 
fingerprint technology that could be used in conjunction with the facial biometric as an 
adjunct or enhancement. 

18. Face recognition algorithms that we had developed earlier for security 
applications were not fast enough to be used in a toy. They typically had about a 3- 
second delay between the image capture and the facial recognition - this delay was too 
long for a toy. Our goal was to make a toy that would recognize and respond to human 
users in real time. 

19. Between September 1996 and February 1997, we began developing 
software for finding, aligning, normalizing and recognizing a unique face, plus detecting 
a not-necessarily-unique facial expression. During this time, using both Visual Basic and 
Visual C++, we began coding algorithms which could locate a face in a digital video 
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image. First, the video signal from a camera was continuously digitized and a number of 
single frame samples were stored in memory (samples that were discarded and replaced 
as the stream continued and computations were completed). The reason we used more 
than one frame was to enable the employment of motion detection/tracking algorithms. It 
was our thinking at the time, that the face would have relative motion to the background 
images. Thus, we utilized a subtraction algorithm which looked at the differences 
between two or more static images and located a region of motion where further analysis 
could be concentrated to detect a face. We also employed neural network algorithms for 
face detection, two dimensional FFTs and Hidden Markov Models. Ultimately, by using 
a blend of algorithms, we successfully solved the problem of finding a face in an image, 
framing it (drawing a circle or box around it) and tracking it as it moves. This legacy 
"box" is still used by the prototype today and can be seen in the demonstration video. 
See Exhibit u 14". A red box indicates non-recognition of the face, while a green box 
indicates recognition. During this timeframe, we also began work on aligning and 
normalizing the located face (making all the detected facial images the same size - pixel 
area - and adjusting for lighting in the image - dark versus light). 

20. Some of the software modules we generated during this time period 
include FR32.exe, FR32a.exe, FR32b.exe, FR32c.exe and CvidCap.dll - iterations of 
algorithms designed for capturing and processing facial images as described in Exhibit 
"8" attached hereto. 

21. During this time frame, we also worked on the integration of a fingerprint 
sensor with an automobile key, as illustrated in the March 8, 1997 notebook entry 
attached as Exhibit "9". The key would have a capacitive sensor integrated in the grip, 
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electronic contacts and an electrical receptor which would be part of the automobile's 
ignition. A schematic diagram is provided which shows how the system would be used 
with a vehicle. During this time, work on the Face Recognition toy and the Integrated 
Biometric Key took place in parallel. 

22. By the spring of 1997, we had developed a reliable solution for locating a 
face in near real-time. So our focus shifted to identifying individual features within the 
face to enable us to align the face better with the matching templates (eigenfaces). We 
first utilized neural networks, trained to recognize eyes, noses and lips (mouth) and also 
subsequently tested "eigenfeatures" for performing this localization. The purpose of this 
part of the algorithm was to locate each feature and find its position relative to the face; 
this enabled us to normalize for scale, translation (position) and rotation. It also helped 
the software determine if the image was actually a face (e.g., a face should have two eyes 
relatively above one nose relatively above one mouth). 

23. Some of the software modules we generated during this time period 
include FR32d.exe, and VideoCap.exe - improvements which provided near real-time 
performance as described in Exhibit "8" attached hereto. 

24. At this step, feature recognition was being accomplished and the computer 
could ascertain a person's emotive facial expressions with no further processing. But at 
this point, the algorithm still has not uniquely identified the person. In other words, 
additional processing was necessary, even after facial expression recognition, in order to 
provide biometric identification. 

25. It must be understood that unique facial recognition is entirely 
distinguishable, both in the object sought to be achieved and in the methods used to 
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achieve it, from detecting a facial expression. Indeed, in our experience, unique facial 
image recognition is significantly more complex than detecting a not-necessarily-unique 
facial expression. Identification of the features only tells us that we have a face, maybe a 
smiling face or a frowning face. But we cannot know who's face we have until we 
perform the next steps in the algorithms of recognizing the combined set of 
features/expressions in the context of the whole (or Gestalt) face. This is an important 
distinction over Kamiya et al., which essentially stopped at the step of ascertaining a 
facial expression (and did not suggest or teach subsequent facial image recognition). 

26. By the summer 1997, we had developed adequate algorithms to locate, 
track, align and normalize the faces. So our focus shifted to performing actual biometric 
identification. 

27. One of the software modules we generated during this time period was 
FR32e.exe - an updated and improved algorithm which could identify facial features as 
described herein above and in Exhibit "8" attached hereto. 

28. Biometric identification, which again is completely different from 
feature/expression recognition, involves treating the entire face as a unique collection of 
unique features that are specific to each given individual, i.e., the distance between the 
eyes, nose and mouth coupled with the morphology of each feature, make up a unique 
identification set much like the ridges in a fingerprint. 

29. Our work centered on perfecting the "eigenface" approach (known 
formally as Principal Component Analysis). The eigenfaces represent orthogonal 
dimensions which collectively describe something we called "face space". 
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30. The first step in using the eigenface approach is to create this "basis" set 
of eigenfaces. We accomplished this task by first obtaining digital images from several 
hundred faces then culling through the photos to identify a "spanning" set of training 
images. By spanning we mean a set of images that contain as many of the feature 
variations we expect to encounter with the faces that will subsequently be recognized by 
the toy. Next, an average face is derived. Then, a standard method called the 
"Karhunen-Loeve Transform" is utilized to produce the actual eigenfaces. 

31. By employing the eigenface technique, the complete face (unique 
collection of unique features), can be reduced to a small set of numerical "coefficients" 
that represent the location of that unique face in face-space. Thus, if six eigenfaces are 
employed, there will be six coefficients representing that face. Recognition is then 
performed by computing the Euclidean distance between subsequent measurements in 
face-space, e.g., the unique coefficients for a first face can be compared to the unique 
coefficients of a second face by measuring the distance between the two face's respective 
locations in face-space. 

32. With the algorithm now working, we next began the tedious effort of 
tweaking parameters (how many faces in the training set, which faces in the training set, 
how many coefficients, etc.) to optimize performance. 

33. Attached as Exhibit "10" is a notebook entry dated September 11, 1997. 
This second reference to the Integrated Biometric Key describes the use of a DTMF tone 
decoder to control the vehicle's ignition upon receiving a recognition signal. This tone 
decoder technology is very similar to that which was utilized with controlling the 
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animation features of the toy, and in this case the two different inventions shared similar 
hardware resources. 

34. During the fall of 1997, we focused exclusively on optimizing the 
algorithm and experimenting with variations to improve performance. We learned during 
this time that we could reliably extract the facial expression information in combination 
with the biometric information to make determinations such as "Bob is frowning", and 
"Sally is happy". It was further discovered that the algorithms could easily recognize 
toys and inanimate objects as long as they also possessed a unique collection of unique 
features that had facial characteristics. We were able to get the technology to identify 
specific dolls as well as human users. Since inanimate objects are incapable of producing 
spontaneous facial expressions, this is another example of the differences between this 
invention and the prior art. None of the prior art references suggest that the facial image 
of a doll could ever be recognized because the prior art does not complete this biometric 
recognition step. By contrast, this invention can and does recognize inanimate (no facial 
expressions) objects. 

35. One of the software modules we generated during this time period was 
Tone.wav - an early file used in testing and developing the use of audio control signals of 
an external device as described in Exhibit "8" attached hereto. 

36. During the few first months of 1998, Dr. Xu and I worked on several 
different biometric inventions at the same time, including a fingerprint-key biometric 
system for which we filed a separate application for patent in 1999. All of our biometric 
projects had significant technological aspects in common with the toy project we were 
working on. 
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37. Attached as Exhibit "11" are photographs in a patent notebook entry dated 
January 5, 1998, of actual hardware built for integrating the Biometric Key with an 
Oldsmobile Silhouette. The system utilized a laptop computer which generated an audio 
signal upon recognizing the user. This audio signal controlled a relay which started the 
engine of the car when energized. Similar hardware was used for the face recognition 
toy. This entry shows the parallels in hardware between the two inventions which were 
being developed simultaneously. 

38. Attached as Exhibit "12" is a patent notebook entry dated March 21, 1998, 
showing excerpts of C++ software utilized in implementing the key system. We used 
similar modules for the toy project. There were several shared functions between the 
systems, especially as described above with respect to the audio signals controlling the 
hardware subsequent to the recognition process. 

39. By the summer of 1998, our work associated with acquiring live facial 
images and processing and recognizing them was largely complete. Our focus shifted to 
developing a working toy demonstration which would use an off-the-shelf platform 
(Teddy Ruxpin was chosen) that could speak, articulate, recognize and interact with a 
human user. In addition to the ability to recognize faces, additional neural networks were 
developed that permitted the toy to recognize numbers, letters and shapes. That way, the 
toy could be utilized for various learning applications. For example, the toy would know 
that it is playing with "Billy" who is six years old and learning shapes. If Billy drew a 
box, it could say "nice job Billy". If Sally came in the room, the toy would immediately 
recognize her and now realize that it was playing with two children and modify its 
interactive behavior accordingly. 
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40. Most of our time from June to August of 1998 was spent in understanding 
the control method for the toy. We used two tracks of audio (left and right). One track 
produced the audio (speech) you hear from the toy, and the other controlled the 
movement of the mouth and eyes. 

41. I had to develop a multi-media workstation that enabled me to record my 
voice along with a control signal (produced by moving a joystick) in time with my 
speaking so that the bear could be properly animated. Next, we stored the combined 
audio and control signals in a .wav file which was later played by the software when the 
toy was interacting with a human user. The .wav file selection was done using decision 
tree logic. This was a difficult and time consuming task. From August through October 
of 1998 we developed most of the demo media scripts using this workstation. By the end 
of October, the first "bench-top" prototype system was ready for integration. 

42. With the earliest prototype beginning to function as intended, we made 
various refinements between October 1998 and December 1998, including incorporating 
the camera into one of the bears. (Most demos are done with an externally mounted 
camera as seen in the video). In addition, we obtained newer model Ruxpins that were 
more reliable in their operation. 

43. By early December, 1998, we finally completed a working prototype of a 
stuffed bear toy with facial biometric recognition capabilities and which provided 
entertaining interaction in response to recognition. By this time, we had developed and 
successfully tested the following functionality, including: (1) the ability to locate and 
track a face in a video image; (2) the ability to scale, normalize, rotate and translate a 
facial image; (3) the ability to recognize and locate facial features (eyes, nose, mouth) 
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within the face further refining the alignment of the face sufficient for Principal 
Component Analysis, and (4) biometrically recognizing individual faces which consisted 
of a unique collection of unique features for a number of different individuals playing 
with the toy. In addition, we integrated the toy with a PC through the use of the sound 
card that enabled articulated interaction with the human user through the use of .wav files 
which consisted of an audio track and control track. 

44. Some of the software modules we generated during this time period 
include Bubba.wav, Dolly. wav, Happy.wav, Sad.wav, and Woody.wav - animation and 
script files used to create the toy interaction as described in Exhibit "8" attached hereto, 

45. Attached as Exhibit "13" is a December 20, 1998 entry that outlines the 
ideas which we incorporated in the Face Recognition Toy Provisional Patent that we filed 
in January, 1999. It discloses the bear, the animation and techniques for accomplishing 
it, and the various games and applications that can be implemented utilizing the facial 
biometric technology. All of the elements of the present invention are discussed in this 
entry, and at this time or very shortly thereafter, a final working prototype was produced 
that functioned very well and demonstrated a complete reduction to practice. A video of 
one of these demonstrations was made and is available on the CD attached as Exhibit 
"14". All demonstrations of the toy to outside parties were made after the filing of the 
provisional application. 

46. We began working on drafting the provisional application in early January 
1999. We filed the provisional application on Jan. 19, 1999. 
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47. I acknowledge that willful false statements and the like are punishable by 
fine or imprisonment, or both, and may jeopardize the validity of the application or any 
patent issuing thereon. 

48. I hereby declare that all statements made of my knowledge herein are true 
and that all statements made on information and belief ace believed to be true. 



STATE OF FLORIDA § 
COUNTY OfQ^|)^ § 

BEFORE ME, the undersigned authority, on this day personally appeared DAVID 
M TUMEY, known to me to be the person of that name, who signed the foregoing 
instrument, and acknowledged the same to be his free act and deed. 

GIVEN under my hand and seal of office this (x_ day of March, 2005. 






fP^k JOHNSON 
£j|JSl MVC0MW»SS*0N # CC 327906 
^IFfSlF June 10. 2006 



Printed Name of Notary 



Commission Expires ^'^ 10 
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AFFIDAVIT OF TIANNING XU 

I, Dr. Tianning Xu, being duly sworn, state as follows: 

1 . I am over 2 1 years of age and am competent to make this affidavit. 

2. I have read Dr. Tumey's affidavit and hereby affirm that the statements 
made therein are true and consistent with my own knowledge, beliefs, and recollection. 




Dr. Tianning Xu 



STATE OF TEXAS § 
COUNTY OF BEXAR § 



BEFORE ME, the undersigned authority, on this day personally appeared DR. 
TIANNING XU, known to me to be the person of that name, who signed the foregoing 
instrument, and acknowledged the same to be his free act and deed. 

GIVEN under my hand and seal of office this day of March, 2005. 



L COURTNEY D.OSGOOD 

NOTARY PUBLIC 
STATE OF TEXAS 

SFc*^^ My Comm. Exp. 05-06-06 
acAaexxocoaooooa"'"'- * 




Notary Publi 



tinted Name <X Notary^/ 



Printed Name <n Notary c 
Commission Expires /OS^^Olp 



Page 14 of 14 



.2/ N<W 



r *7 





s 



ne^^ivv^r ******* 




CI*** 



lb 1 - 






Mi***, 



J ! 





(5> S^rtf- C***> 



^ ^^^^ ftfrtJb- &A<*4 etc?***** - 



4/< 



5> £ AVSS"***- *&pu^ 




5# TU-i^T Cev^fi-^ 




MUX in IrSL tydc* TV* 6^»s5«^ A^^ 1 "- 




i i 




t i 



ft) yy^t^nrtp 



i i ! 



i i 




d ^HH^H ^-i Mill ' 




i7 





!- I 



iii 




. . ^> :;:.:;. 

: : ^ i : : M : M 



\ 

i 




: )■ i i 



! i i i 



i i • j i 




\ PnuJLi w£*J> < 

. lir^tA-ft A ; (2, \ t ; . , 5; 



6 > 



ft'fr. 





V - 



j r i i : : ! : 1 * ' ' : 




. I- .-( I 



•j ■ ( ! 



1 I 




St 



; .H. 



i I i 1 I : ■ \ ' ' 

\ ■! i = ' 1 



I . ; i 



i j 1 i ; ' i : '. J^---' 



I ! 



-I" ; 



J; 




•A &^£,< Hnfj9: ^>H^ 





















■ ' i 










i i 



■[ ■ 
t i 



! ' ! j 

i , \ 

r - i 1 
i 



}^f v y ^ i 
: ,iM^h \p*»h>* 




•i 2r^ c 



i i 



T ! 



* k 





I Ai 



1 .1, ^ -1-1 




t 



2z 

1A * 



4 - 6^)(^r^ui^ ^i* 12 ^)., 



0 w 



^ 1 Al; 





f 




O \ 2, 3 H ' S"U» 



0 

1 

L 
> 

A 




3/ 3> 




■ i 




Fity^ ■ fy/kiatv^ ; \*/v 




:,'<*'..'.. ..... 

<g) y^- , s "^' 




jT 



-fir \Z>n*^ 



r , 





— A^^- -r COvwaM^; .t 1 ^;^- S--r^te»- isn^<^ i'ov r 1 -* : ~t*A< , 



iSiV (-Jet 12. « X (2g 'p/x— ejP I i~^jta 

; - i . ; . 



<9 /U*rf^l I>S3-iu 3 
-p/ci" ... . 

2 




I , ; 

*>"2 j 




i i 




! ? ! ! 



4- t^Of^' 



! ! 



*0! 





; ■ 










6 



i p d ^ iWv : : j . . : ; I . 

■ - < ' ! ■ ' '■ . 

■<j£ \ Si. \lA j [ C + *p ' : 



: i|p ( L ;^:^uu; 4 **** ; 



1 i 




P. 

0- 




\ \ § ■ o f : ! p**j >a*> i^ 1 - ; : : : 



; : ■ • i 1 ■ ; • 1 ; - i •• i 

, .; ! ; ; ;. i. . i i i * ■ . j . 



T 



. \ 



> 




V^V ; 



1 



J 



pit* 




V 



1 



: ■ ■ *^f<j> fe6lv -l 



fix?-- 



Nor , 



tl: 



v 









Kir 


: 



-pi. If 








K : : : 



n 



~x>y6 






g0t^l~JL46>Ci*~^ WtA- \A-*>. flee****, t~ii*h> ; hH*yp, 



3r 



1 i- i 




1 f ! 



.1 



i ! ! 



i I' 




AM 



i - ! 



f 



i 4 



. 'i I 




I ' 1 



IVS Face Recognition Toy Software Development 
Work Product and Chronology 
2/14/05 

The CD attached to this document contains the work product (software programs, files & 
data) of the Face Recognition Toy from 1996 through 1999. The files are stored in 
folders by year: IVS 1996, IVS 1997, IVS 1998, IVS 1999. Each folder contains files 
generated in their respective calendar-year. Each file has its original date of last 
modification. To view the dates, right-click on any file and select "properties". The 
Creation Date is the day the file was copied to the CD and thus 'created'. The Modified 
Date is the day the program was last modified and is the same as the original 'creation' 
date for the file. The Accessed Date is today's date, the date the file was last viewed or 
opened. 



Folder: IVS1996 



File Name 


Date of Creation 


Function 


IVSFace.exe 


7/15/96 


Software for finding, aligning, 
normalizing. Also able to match 
a face with one in a database. (No 
real-time capability). Note: This 
work predates the development of 
the Toy algorithms but was the 
foundation upon which other 
algorithms were developed 


FR32.exe 


9/4/96 


Uses feature recognition (i.e., 
expressions) to locate Eyes and 
align face with an optimal 
matching template. First in the 
series of algorithms developed 
specifically for Toy application. 


FR32a.exe 


9/14/96 


Updated version ofFR32.exe 


FR32b.exe 


11/1/96 


Updated version of FR32a.exe 
(Semi-automatic not quite real- 
time capability). 


IVS.dbs 


8/17/96 


Database of Eigenface 
coefficients used in early S/W 
development. 


_isreg32.dll 


4/29/96 


Dynamic Link Library (DLL) 
used by above executables. 


Cvidcap.dll 


12/28/96 


Dynamic Link Library (DLL) 
used for capturing and tracking a 
face in a video image. 


Reader32.dll 


3/5/96 


Dynamic Link Library (DLL) 
used by above executables. 


Images (Folder) 


6/10/96 


Original spanning set of fecial 
images utilized in training Neural 
Network algorithms and creating 
Eigenfaces & Eigenfeatures. 



Folder: IVS1997 



rue l^uinc 


Tlfitp nf Creation 


Function 


FR32c.exe 


1/9/97 


Updated version ofFR32b.exe. 


FR32d.exe 


6/21/97 


Near Real-Time automatic face 
finding & matching, upaatea 
version ofFR32c.exe 


FR32e.exe 


7/27/97 


Updated version of FR32d.exe 


VideoCap.exe 


5/15/97 


Software for locating and 
capturing a face in a video image. 
See example file Vcap.bmp 
below. 


DataO - Data9 


1/8/97 


Database of Eigenface 
coefficients used in early S/W 
development. 


Diskl -Disk3.zip 


6/29/97 


Installation package for 
FR32x.exe - used to install the 
software for use in prototype 
development. Files zipped 
created 6/24/97. 


Express.dbs 


1/8/97 


Facial expression database - early 
work to create Eigenfeatures to 
recognize smile, frown, etc. 


Tone.wav 


10/14/97 


Tone.wav is an early file used in 
testing and developing the use of 
audio 'control' of an external 
device (Toy). A DTMF tone file. 


Vcap.bmp 


5/27/97 


A facial image captured and 
stored using VideoCap.exe. 
Notice a great photo of the still 
"young" inventor. 



Folder: IVS1998 



File Name 


Date of Creation 


Function 


1.ICO-11.ICO 


11/18/98 


Icon files of Teddy Bears, used in 
developing the software for the 
Toy. Icons were used with 
program features such as buttons, 
screens & graphics. 


Bubba.wav 


10/22/98 


Animation Script & Control .wav 
file. Notice one track of audio 
and one track of tone-control 
signals. This file was used when 
a toy animal was recognized. 


Dolly.wav 


10/22/98 


Animation Script & Control .wav 
file. Notice one track of audio 
and one track of tone-control 
signals. This file was used when 
a toy doll was recognized. 


Happy.wav 


10/22/98 


Animation Script & Control .wav 







file. Notice one track of audio 
and one track of tone-control 
signals. This file was used when 
a happy face (smiling subject) 
was seen by the toy (after 
recognition) 


Sad, wav 


10/99/08 


Animation Script & Control .wav 
file. Notice one track of audio 
and one track of tone-control 
signals. This file was used when 
a sad face (frowning subject) was 
seen by the toy (after 
recognition). 


Woody.wav 


10/22/98 


Animation Script & Control .wav 
file. Notice one track of audio 
and one track of tone-control 
signals. This file was used when 
a toy action-figure was 
recognized. 


Folder: IVS1999 


FUe Name 


Date of Creation 


Function 


FaceKey2.exe 


2/23/99 


Early compilation of the final 
configuration for the Toy Face 
Recognition Software. Almost 
identical to the software used in 
the current Toy demonstration (as 
seen in the video). 


Bubba.pk 


7/30/99 


PK amplitude file generated by 
the audio workstation to 
automatically adjust volume 
levels. 


Dad.wav 


7/28/99 


.wav file played when the toy 
recognized the inventor (used in 
demonstrations and can be heard 
in the video). 


Herewego.wav 


7/28/99 


An interactive .wav file used 
while Jacob was surfing the 
internet with the Toy acting as a 
'Web Nanny'. 


Jacob.wav/Jacob.pk 


7/28/99 


.wav file played when the toy 
recognized the inventor's son 
Jacob. 

Note: The audio refers to Jacob 
as being "3 years old". Jacob 
was born in 1995 corroborating 
this "Jacob " wave file was 
created prior to Sept J 999. 


Jacob 1 .wav/Jacobl .pk 


7/26/99 


An interactive .wav file used 
while Jacob was surfing the 
internet with the Toy acting as a 
'Web Nanny'. 



r 



Jonathan. wav/Jonathan.pk 


7/28/99 


.wav file played when the toy 
recognized the inventor's son 
Jonathan. 

Note: The audio refers to 
Jonathan as being "6 years 
old". Jonathan was born in 
1992 corroborating this 
"Jonathan " wave file was 
created prior to Nov 1999. 


Jonathanl .wav/Jonathanl .pk 


7/28/99 


An interactive .wav file used 
while Jonathan was surfing the 
internet with the Toy acting as a 
'Web Nanny'. 


Squeeze.wav/Squeeze.pk 


7/28/99 


An interactive .wav file used to 
control the actions of the internet 
after the subject had been 
recognized. 
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wstr = new WCHAR[len]; 
// CP_ACP 

MuluByteToWideChar (CP_OEMCP, 0, s, -I 
#endif 
} 

AtoWConverter :~AloWConverter 0 { 
delete [] wstr, 

> 

Ato WConverter :operator LPCWSTR 0 { 
wstr, 



wstr, len); 



} 

WtoAConverter :WtoAConverter (LPCWSTR w) { 
#ifflef UNICODE 

// In this case bath strings are Unicode and we just use normal tetrcpy. 
TCHAR* _astr = new TCHAR[lstrIen(s)]; 
return lstrcpy (_astr, s); 
#else 

// We need to convert the Unicode to ASCII string and copy after that 
// CP_ACP 

int len = WideCharToMuhiByte (CP_OEMCP, 0, w, -1, NULL, 0, NULL, NULL); 
astr = new charflen]; 



// ClassWizard generated virtual function overrides 

//( { AFX_VIRTUAL(C AboutDlg) 

protected: 

virtual void DoDataExchange(CDataExchange* pDX); 
//}}AFX_ VIRTUAL 

// Implementation 
protected: 

//{ (AFX MSG(CAboutDlg) 

//}}AFX_MSG 

DECLARE MESSAGE_MAP() 

}; 



CAboutDlg::CAboutDlgQ 



// CP_ACP 

WideCharToMuhiByte (CP_OEMCP, 0, w, -1, 
#endif 
> 



astr, len, NULL, NULL); 



//{ { AFX DAT A_ENIT(C AboutDlg) 
//} } AFX_DATA_INIT 



void CAbc^Ig::DoDataExchange(CDataExchaiige* pDX) 



WtoAConverter::- 
delete [J astr, 

} 



-WtoAConverter 0{ 



WtoAConverter :operator LPCTSTR 0 { 
return astr, 



//Global variables 



int gnWidth, gnHeight; 
FT HANDLE g_FtContext; 
FT HANDLE gdbContext; 
DB USERID g currentUserlD; 
FT_UI_LINK gFtUILtnk; 
FT_DEVICE_INFO g Devlnfo; 
FTBYTE* gFeatures; 





CDialog: :DoDataExchange(pDX); 
//{ { AFXDAT AM AP(C AboutDlg) 
//} } AFX DATA MAP 



BEGIN_MESSAGE_MAP(CAboutDIg, CDialog) 

//{ ( AFX MSG MAP(C AboutDlg) 
// No message handlers 

//}}AEX_MSG_MAP 
END MESSAGE MAPO 

luiiuiiiuiiiiiiiiiiiiifiiiifuiiiiuiiituiiiiiiuintiiiimuiniaimi 

II CFTSampleDBDlg dialog 



//width and height of drawing area 
//dpFpFns context 
//dpDbase context 
//current user ID 
//UI link information 
//device info 
//features 

int g_recommendedFtrLen, g minFtrLen; //recoiranended features length and minimi 
int gjiFmgerCount; //variable to count down fingers left while registering 

BOOL g bkVerifying; //verifying or register mode, in verifying mode do 

CRect grectDrawArea; //rectangle to draw fingerprints in dialog 

CWmThread* g_pRegisterThread = NULL; //pointer to register thread 
CWinThread* gj)VerifyThread =* NULL; //pointer to verify thread 

BYTE *g FPBuffer, // Pointer to buffer which holds data portion of bttn void CFTSampleDBDlg: : DoDataExchange(CDaUExchange* pDX) 

// This pointer is provided for toolkit in the callback function. { 

// Toolkit loa ds image in the buffer and notifies GUI thread th 
HWND g_hwnd = NULL; //HWND of dialog 

BOOL g blsTimeToDie = FALSE; //variable used to ensure clean exit from pr 
CEvent g_event(FALSE, TRUE); //event used to make sure that program will r 

//while toolkit is doing something (like watting for image) 

//second argument TRUE for manual CEvent 



CFTSarnpleDBDlg::CFTSampleDBDl^CWnd* pParent /*=NULL*/) 
: CDialog(CFTSampleDBDlg::IDD, pParent) 

//{ {AFX DATA INIT(CFTSampleDBDIg) 
m_strPrompt = _T(""); 
//} } AFX DAT A INIT 

// Note that Loadlcon does not require a subsequent Destroylcon in Win32 
m_hIcon «= AficGetAppO>LoadIcon(IDR MAINFRAME); 



CDialog::DoDataExchange(pDX); 
//{ { AFX_DATA_MAP(CFTSampleDBDIg) 
DDX_Text(pDX, IDC EDIT PROMPT, m_strPrompt); 
//} } AFX DATA MAP 



BEGIN MESSAGE MAP(CrTSampleDBDlg, CDialog) 
// { { AFX_MSG_MAP(CFTSampIeDBDlg) 
ON WM SYSCOMMANDQ 
ON_WM_PAINT0 
ON WM QUERYDRAGICONO 

ON BN_CUCKED(IDC RADIO REGISTER, OnRadioRegister) 
ON BN_CUCKED(IDC RADIO VERIFY, OnRadio Verify) 
X>N WM DESTROYQ 



r 



i L. 



switch(pStatiis->code) retumO " " " ~ 

case FT READY TO FILL_BUF: ' 

SendM^ge(g„hv^ MKS AGE R^Y_TO_FILL_Bl ^ ^VerifyCLPVOID pParam) 
TRACE( FT_READY_TO_FILL_BUF\n ); ^ 

break; HWNDhwnd; 

DB USERID userlD; 



//verify thread function 



case FT BUF FILLED: 



DB FINGER KEY fingerKey; 



SaidMessagefe Jtwnd, MESSAGE IMAGERECEIVED, 0 - RETCO r)E rc 
TRACE(-FT.BUF„FILLED\n7, fTvERJSCORE score; 

break; FT BOOL bis Verified; 

FT RESULT result* 

caseFTJMAGEJNFO: // ^ IM^UAUTY w{ 7„ar name[DB USERNAME LEN+1]; 
SendMessage(g hwnd, MESSAGE_FT_IMAGE_INFO, 0, (. pr i MG QUALITY qualhylmg; ~ 



TRACE("FT_IMAGE_INFOV); 
break; 

case FT FEATURES INFO: 
SendMessage(gJrwnd, MESSAGE FT FEATURES_INFO : 
TRACE("FT_FEATURES_INFO\n ,t ); 
break; 

case FT WATTING FOR IMAGE: 
SendMessage(g_hwnd, MESSAGE FT WATTING FOR IN 
TRACE("FT_WATnNG_FOR_IMAGE\n M ); 
break; 

default: 
break; 



FT FTR QUALITY aualityFtr, 
wWle(!gJ>IsTimeToI>ie) 



{ 



> 



hwnd = : :GetDlgItein((H WND)pParam, IDC EDIT PROMPT); 

: : Set Windo wText(hwnd, "Please put your finger on sensor"); 

DB JopUser(g_dbContext, AuseriD); 

gevent ResetE vent(); 

rc - FT_acauh«Featuiies(g_FtContext, 

FTVERFTR, 

g^recommendedFtrLen, 

gFeatures, 

&qualitylmg, 

&qualityFtr, 

&resuhX 
if(result == FT SUCCESS) 



-j- return FT ID CONTINUE; 



{ 



} 

FT DIS PLAY BUF PT getRegisterDispIayBuf (FT_FTR_TYPE, vc 
{ 

TRACE("getRegisterDispIayBuf\n n ); 
return gFPBuffer, 

} 



gevent.SetEventO; 

hwnd - ::GetDlgIlem((HWND)pParam, IDC EDIT USERNAME); 
::SetWindowText(hwnd, "Unknown user"); 
hwnd = : :GetDlgIten<(HWND)pParam, IDC EDIT FINGER); 
::SetWindowTexu>vnd, nn ); 
while(DB_nextUser(g_dbContext, userlD, &userID, 
NULL, NULL, NULL, NULL, NULL) != FT WRN EOF) //loop through the list of users 



{ 



i FT RETCODE releaseRegisterDisplayBuf (FT DISPLAY BUF PT 



TRAC^ w rdeaseRegtsterDisplayBui>n , 'X 
return 0; 



//Thread functions 

UINT DoRegister(LPVOID pParam) 
{ 



//register thread function 



g_nFingerCount - 0; 
HWNDhwnd; 

hwnd - ::GetDlgItem((HWND)pParam, IDC EDIT PROMPT); 
: :Se* WindowTexuTiwnd, "4 fingers left**); 
FT RETCODE nr, 
FT RESULT result; 

g_e vent ResetEventO, //we reset CEvent in nonsignaled su 

//program will wait until FT register comple 
rc = FT_register(g_FtContext, 

FT TRUE, //allowLeaming has to be FT TRUE 

//use features with database 
greoofnmendedFtrLen, 
g^Features, 
NULL, 
&result); 

g event SctEvenl(); //now wc can exit 

::PostMessage((HWND)pParam, WM REGISTERTHREADFINISHED, 0, (LPARAM)resuh); 



g event ResetEventO; 

rc - DB authenticate^ _dbContext, userlD, NULL, DB UNKNOWN, g Features, NULL, 

NULL, &fmgerKey, &score, NULL, &bls Verified); 
g^eventSetEventO; 
iflblsVerified) 
{ 

geventResetEventO; 

DB readUserData(g dbContexl, userlD, name, NULL, NULL, NULL, NULL); 
geventSetEventO; 

hwnd = ::GetDlgh^(HWND)pParain, IDC EDIT USERNAME); 
: :Set WiiKk)wText(hwnd, WtoAConverter(name)); 
PostMessage((HWND)pParam, WM_USER VERIFIED, 0, (lJ > ARAM)fingerKeyiinger); 



> 

} 

} 

else 
{ 

g_eventSetEventO; 

: :PostMessage((HWND)pParam, WM VERIFYTHREADFINISHED, 0, 0); 
return 0; 

> 

} 

::PostMessage((HWND)pParam, WM_ VERIFYTHREADFINISHED, 0, 0); 
return 0; 



} 
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□ black borders 

□ image cut off at top, bottom or sides 

□ faded text or drawing 

□ blurred or illegible text or drawing 

□ skewed/slanted images 

□ color or black and white photographs 

□ gray scale documents 

□ lines or marks on original document 

□ reference(s) or exhibit(s) submitted are poor quality 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



